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ABSTRACT 


The National Preparedness Goal directs that preparedness efforts for major events 
should be developed through a capability-based planning process. The ability to perforin 
a valid and reliable capability assessment is necessary to guide proper allocation of 
resources in the capability-based planning process. Capability assessment is also 
necessary to validate and measure the impact of investments. This thesis examines a self- 
assessment methodology that may be used to perform capability assessment. A 
comparative case study methodology is employed to examine process options. These 
options might be utilized to detennine which process provides the most opportunity for 
data collection while fostering a collaborative environment that may provide 
opportunities for sharing knowledge among assessment group members. A review of the 
case studies suggests that the subject-matter experts employed in the analysis may not 
have the requisite knowledge to perform the assessment. This suggested finding implies 
problems with capability-based planning efforts. The thesis makes recommendations for 
future strategies to perform homeland security capability assessments. 
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EXECUTIVE SUMMARY 


To meet the requirements of HSPD 8, DHS has proposed a self-assessment 
process to perform the national capability assessment. This document examines the 
results from a test deployment of the proposed system. The expected results of the 
deployment were the production of data that could be used to measure the effect of 
investments in homeland security to provide a snapshot of preparedness. DHS also 
expected this methodology to enhance the knowledge of participants in the assessment 
process. 

The results of the comparative case study suggest a flaw with a basic assumption 
of the assessment process. The study suggests that the locally appointed subject-matter 
experts may not be able to employ the proposed methodology to produce valid and 
reliable data. The study did show an undercurrent of preparedness activities that were not 
captured by the assessment tool, but suggest collaborative homeland security planning 
efforts exist at the local level. 

The findings are significant because they undermine the capability-based planning 
process. Capability-based planning develops investments based on gaps and needs 
identified in an assessment. Faulty assessments will reduce national preparedness by 
causing our decreasing resources to be invested to close gaps that may or may not exist. 

This thesis recommends that DHS take steps that create an understanding of how 
capabilities are implemented at the local level, before efforts are made to assess those 
capabilities. The document also recommends that the National Response Plan, HSPD 5 
and HSPD 8 be coordinated so that requirements and guidance regarding capabilities be 
made consistent. 


xi 
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I. INTRODUCTION 


A. PROBLEM STATEMENT 

The National Preparedness Goal (NPG) as set forth in Homeland Security 
Presidential Directive # 8 (HSPD 8) is a plan to ensure that our country is prepared for a 
catastrophe and positioned to prevent a tragedy. Through the designation of capabilities- 
based planning and universal tasks, the goal sets a path to preparedness. To support the 
goal, federally funded grant programs provide resources and incentive to local 
implementation of the plan. 

A solid foundation is required for any serious undertaking. In capabilities-based 
planning, the foundation is performance of both risk and capabilities assessments. This 
thesis focuses on one of these pillars: the capability assessment. If the capability 
assessment is not adequate, everything that follows is built on a broken foundation. In 
educational policy development the focus on foundations is understood. For example, the 
foundational skills of reading and writing are taught prior to critical analysis of poetry. 

Why then, have we ignored the foundation of our capabilities-based planning 
cycle? The last national capabilities assessment was conducted in 2003, yet we have 
gone on to create and fund implementation of countless homeland security programs. 
These are commendable accomplishments, but — lacking an interwoven capability 
assessment — the accomplishments are built on a poor foundation. Without perfonning 
assessments as part of the investment process, it is unknown how far those investments 
have moved the nation toward meeting preparedness goals. 

My experience with homeland security efforts in the National Capitol Region 
(NCR) demonstrates assessment to be a factor in the success or failure of efforts to 
implement the NPG. The inability to measure the effects of investments that are 
implemented as part of the NPG, hinders the ability to gain funding for programs that 
could bring safety and security. 
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The nation’s previous attempts at assessment did not produce changes in behavior 
or create plans that caused improvements in homeland security. The 2003 National 
Capabilities Assessment was a paper lion, consuming personnel resources and providing 
little useful feedback. 1 

The 2005 and 2006 Homeland Security Grant Application processes required the 
performance of a gaps analysis based on the NPG. In the NCR, the 2005 gaps analysis 
was completed with technical assistance from the Department of Homeland Security. 
This effort resulted in three consecutive days of proctored discussion about existing gaps. 
The focus group participants turned the discussion to defending existing grant funding 
requests. The result of this effort was a report in which identified gaps that aligned with 
funding requests. This effort did not provide data on existing gaps, nor did the 
assessment process produce an environment that encouraged open and fra nk discussion. 2 

This faulty foundation for the regional grant applications was a contributing factor 
in the low marks the submissions received for relating funding priorities to existing gaps 
in the NPG. The ultimate result was reduced funding and reduced ability to improve 
preparedness. 

In 2006, these inefficient top-down processes resulted in the local development of 
a self assessment for capabilities that fell under the responsibility of Emergency Support 
Function #4, Firefighting. This effort was well-received within the discipline, and 
produced data and discussion that helped to create stronger homeland security capability 
assessment. By strengthening the foundation, the product improved. 

Members of Emergency Support Function (ESF) 13, Public Safety and Security 
Annex, were exposed to the self-assessment in their supporting roles to ESF 4. This 
exposure caused them to adopt the self assessment and deploy it for target capabilities in 
which they were the lead agency. Both ESF 4 and ESF 13 felt this self-assessment 


1 GAO Report, 05-121, “Homeland Security, Management of First Responder Grant Programs Has 
Improved, but Challenges Remain,” February 2005, 17 

2 This paragraph is a summary of observations made by the author after participation in the 2005 
Homeland Security Grant Process. The author was participating in the assessment process as the leader of 
the Metropolitan Washington Council of Governments, Fire Chiefs Hazardous Materials Subcommittee. 
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process triggered an enhanced understanding of capability status. This understanding 
resulted from breaking the assessment processes into manageable modules and rapidly 
providing assessment results. Planners and decision makers used the results from the 
assessment to obtain an understanding of the whole capability — not just specific pieces. 

This local assessment solution was unfinished, but it created an environment that 
sought to follow the preparedness cycle and focus on a foundational assessment process 
to inform investments. Members of multiple disciplines exposed to the survey credited 
the survey process with inspiring them to become more familiar with the elements of 
their capability. The results of the survey gave them an understanding of the status of 
their plans and programs in relation to meeting the requirements of the NPG. This 
capability survey effort was a step in the right direction. By providing a definition of 
current capability, our small homeland security community sought cross-discipline 
solutions to gaps — instead of chasing funding for single-discipline programs that 
already had high levels of implementation. 

At the national level, the Target Capabilities Pilot Assessment Working Group 
has been at work developing the next National Capabilities Assessment with similar 
characteristics to the locally developed solution described above. This working group 
developed the Target Capability Self Assessment Tool and process described in this 
thesis. 

B. RESEARCH QUESTION 

A DHS working group has proposed a methodology using locally selected 
subject-matter experts to perfonn capability assessments. Can a portion of this 
assessment methodology, designed to create collaboration, be altered without affecting 
the assessment of capability status? What is the effect of the process changes on the 
desired additional benefits: enabling and improving collaboration among homeland 
security professionals? 

In the proposed assessment methodology, collaborative opportunity is an expected 
byproduct of the assessment process. In this process, an assessment team reaches a 
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consensus opinion regarding a capability status. DHS expects that the consensus process 
will facilitate sharing of information and the creation of relationships among assessment 
team members. The benefits of this process are expected to include more accurate 
assessments and the creation of relationships that support future capability-building 
efforts. What is the effect of different processes on the expected collaborative benefits? 
Does changing the assessment process, from a consensus method to an aggregated 
method, affect the expected benefits of information-sharing and relationship-building? 

C. HYPOTHESIS 

The hypothesis for this research question is that altering the assessment process 
will not play a key role in the results, and that capability status assessments from the 
different processes will agree. A further hypothesis is that of the different processes, one 
process will be identified as having provided more additional benefits than the others. If 
the hypothesis is true, it will result in the ability to identify which process provides the 
greatest opportunity to determine capability status and provide additional benefits to the 
assessment team. 

D. SIGNIFICANCE OF THESIS 

Homeland Security is a nascent discipline, with dedicated guidance and funding 
initially occurring as a result of the Oklahoma City and Tokyo terrorist attacks. 3 The 
terrorist attacks of 9/11 rapidly increased pressure on the discipline to increase national 
levels of preparedness. The maturing process of the discipline requires academic study of 
methods used to evaluate the components of the discipline. To date, there has been some 
research on measuring homeland security preparedness status, but this literature is not 
equal to the task of understanding the intricacies and nuances that occur in the 
performance of homeland security capability assessments. 4 This thesis will expand on 
that body of knowledge by focusing on the processes needed to perform a capabilities 

3 Andy Mitchell, “On Domestic Preparedness Efforts,” Testimony before the Committee on the 
Judiciary, United States Senate, April 20, 1999, http://iudiciarv.senate.gov/oldsite/42099am.htm accessed 
August 16, 2007. 

4 Glen Woodbury, “Measuring Prevention,” Homeland Security Affairs I, no. 1 (Summer 2005), 10. 
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assessment in the homeland security environment. Specifically, the findings in this thesis 
will examine how members of the homeland security discipline are able to perfonn an 
examination of their field. 

Homeland Security practitioners involved in planning activities will benefit from 
the development of a framework for understanding an assessment process proposed for 
use in homeland security. Ultimately, this understanding of the assessment process will 
help decision makers when allocating resources to improve capabilities. 

The findings and recommendations included in this thesis will help national 
leaders understand capability assessment at the local level. Homeland security leadership 
can use the suggested findings presented in this thesis to develop capability assessment 
methodologies that meet both national and local priorities. 

E. LITERATURE REVIEW 

The terrorist attacks of 9/11 and the devastating effects of Hurricane Katrina 
caused the nation to focus energy and resources on how the nation plans for and responds 
to emergencies. The studies, results and ideas are incomplete, but the environment this 
examination has wrought is producing new ways of looking at how the basic functions of 
planning are performed in the emergency preparedness community. This thesis will 
focus on the methodology of measuring capabilities. This is also an area that has caused 
angst for emergency preparedness planners, as they attempt to quantify or classify 
capability. 

The starting point for the literature review is the government documents that state 
the requirement or assess the results of the existing reviews. A capability review is 
required by HSPD 8, Section 24 Assessments and Evaluations. 5 Additional requirements 
for the capability review are found in State Homeland Security Program and Capability 
Review Guidebook, Volume 1. This document defines minimum requirements of the state 


5 George Bush, Homeland Security Presidential Directive/HSPD-8 (Washington, DC: The White 
House, 2004), 5. 
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capability review and how the results are used in the grant process. 6 The accuracy and 
results of the capability review are indicated by the importance placed on them in the 
grant allocation process. The Department of Homeland Security specifically suggests 
that grant applications be used to fund gaps in state capabilities as identified in the 
capability review. 7 The Department of Homeland Security, Grants and Training Office 
directs that scoring of grant applications will be based on the ability to clearly state a plan 
to close weaknesses in state capabilities. 8 Unfortunately, the guidelines to perfonn a 
capability review are not clearly specified in the capability review guidance. 

The NPG is the roadmap proposed by the federal government to create a nation 
prepared to prevent, respond and recover from natural disasters and terrorist attacks. The 
NPG has three components: National Planning Scenarios, the Target Capabilities List and 
the Universal Task List. Within the NPG, capability-based planning is described as the 
preferred method to developing investments to reduce the risk from all types of hazards. 9 

The NPG provides a basis to assess preparedness to set priorities for the most 
effective use of limited resources. 10 This basis is the Target Capability List (TCL). The 
TCL is a list of thirty-six capabilities that state and local governments are expected to 
create and maintain. Not every jurisdiction will be required to maintain each capability; 
the decision on which capabilities to implement should be decided based on the risk and 
circumstances present within an area of hazard. 11 To implement these broadly defined 
capabilities, coordination is needed between government, citizens and the private sector. 


6 Department of Homeland Security, State Homeland Security’ Program and Capability Review 
Guidebook, Volume 1 (Washington, DC: Department of Homeland Security, 2005), 4. 

7 Matt Mayer, letter to Edward Reiskin, October 21, 2005. 

8 Tracie Henke, “Application Review and Scoring Approach for Fiscal Year (FY) 2006 Homeland 
Security Grant Program.” G&T Information Bulletin, no. 202 (February 2006): A-l. 

9 Department of Homeland Security, National Preparedness Goal (Washington, DC: Department of 
Homeland Security, 2005), 4. 

10 Department of Homeland Security, Target Capabilities List, Version 2.0 (Draft) (Washington, DC: 
Department of Homeland Security, 2005), 5. 

11 Department of Homeland Security, National Preparedness Goal (Washington, DC: Department of 
Homeland Security, 2005), 7. 
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Amy Donahue points out controversy in the effectiveness of the TCL as a 
measuring tool. She argues that the TCL is too large and that it assumes common 
behaviors across emergency response disciplines. As a result of this generality, 
capability requirements may vary across different target capabilities and obscure the need 
to focus on local coordination and shared capabilities. 12 

Current work by DHS is focused on formulating strategy to conduct capability 
assessments that are systemic, repeatable, useful and consistent in what they measure and 
the benefits the assessment produce. The assessment goals are twofold: to gather data on 
the nation’s level of preparedness and to facilitate local understanding of preparedness. In 
Florida, a recent pilot assessment identifies issues that should be addressed in order for a 
capability assessment to help regions develop programs that are consistent with strategic 
plans and provide a database to measure the value of investments in those programs and 
plans over time. 13 The report is implying that an assessment tool alone cannot create the 
opportunity to infer collaboration between disciplines, and the process for utilizing that 
tool is as important as the tool itself in the final use of the assessment data. 

The ability to measure collaboration is of equal importance because accurate 
measures of these complex capabilities require multiple sources of information from all 
participants. The failure to include one agency in the assessment process could lead to an 
inaccurate assessment. In this manner, each capability requirement must work together 
with the others to equal a capability that is more than the sum of the separate parts. The 
extent to which these separate parts work together to achieve the greater output is 
collaboration. Understanding the collaboration effort is therefore important to 
understanding the impact of the measured capabilities on preparedness. 

Any method used in the emergency preparedness community could usefully be 
compared to established capability or program assessment systems used elsewhere in 
government to assure wide acceptance. The Program Evaluation Profile tool, developed 

12 Amy Donahue, Concerns about the Department of Homeland Security’s Target Capabilities List 
Instrument and Process (talking points paper presented to Interagency Board Chair, DRAFT, June 2006). 

13 Department of Homeland Security, Discussion on Changes to the Pilot Capabilities Self Assessment 
Tool Pilot Capabilities Assessment: Methodology Meeting (Washington, DC: Department of Homeland 
Security, October 25, 2006), slides 10, 11. 
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by the Department of Labor, is a system for measuring safety programs in an effort to 
make the workplace safe. Because this tool seeks similar results to the emergency 
preparedness field, any similarities will provide an area of agreement within government. 
There are other standardized evaluation tools and methodologies in the Government 
Accounting Office and Office of Management and Budget (OMB). OMB uses the 
Program Assessment Rating Tool (PART) which also uses an assessment tool. 

Theodore Poister and John Thomas, of Georgia State University, examine the 
possibility of using the methodology from James Surowieki’s “Wisdom of Crowds” to 
assess citizen perception of the highway department. In their presentation “The 
‘Wisdom of Crowds’: Learning from Administrators’ Predictions of Citizen Perceptions,” 
the authors examine two case studies. They conclude that the theory has merit in 
discovering unknown information and is suited to specific types of crowds. The concept 
of discovering unknown information has parallels to discovering whether collaboration is 
occurring within a target capability. The lessons from this process will be addressed in 
the thesis. The small size of this comparative case study is problematic, but it is of 
interest because the findings are similar to the predictions made by Surowieki. 14 

Recent literature examines measurements of homeland security capabilities. Glen 
Woodbury’s article, “Measuring Preparedness”, advocates an approach at the state 
government level to measure preparedness outputs. Woodbury’s scale applies broad 
questions about prevention outcomes to determine critical gaps in knowledge about the 
effort being applied to close those gaps. He argues that this method can be used in the 
prevention field and to predict consequence. The assessment tool he uses is similar to the 
one proposed in this thesis, but the methodology of his assessment is different. 15 

The literature relating to collaboration in homeland security is more diverse. In 
their report on building collaborative capacity, Thomas, Hocevar and Jansen describe the 
benefits of collaboration as cost savings, improved decision making and maximizing the 

14 Theodore Poister and John Thomas, “The ‘Wisdom of Crowds’: Learning from Administrators’ 
Predictions of Citizen Perceptions” (presentation at the Public Management Research Conference, Los 
Angeles, CA, September 29, 2005), 21. 

15 Glen Woodbury, “Measuring Prevention,” Homeland Security Affairs I, no. 1 (Summer 2005): 10, 
http://www.hsaj.org/7articleM.L7 (accessed September 13, 2006). 
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efficiency of scarce resources. The authors argue that improving collaboration is 
especially important in instances when organizations must work together in support of a 
common goal . 16 The benefits of collaboration that Thomas, Hocevar and Jansen have 
identified will need to be examined as effective measures of collaboration in the 
homeland security community. It may be possible that collaboration exists, but it does 
not produce the expected results, or that no collaboration exists and the identified outputs 
are met. 

A Government Accountability Office (GAO) report examines collaborative 
efforts in the federal government. The report identifies areas that can be examined to 
assess collaboration. These areas are: definition of common outcome, joint strategies, 
leveraging resources, defined roles and policies, and accountability . 17 The GAO issued a 
follow-up report that found the government’s efforts at homeland security collaboration 
to be at a high risk for failure . 18 The definition and impact of this failure will need to be 
examined. There may be instances where a lack of collaboration is acceptable to the 
parties involved, and the capability levels achieved by the parties are acceptable to their 
communities. 

The primary gap in the literature is that DHS has yet to publish a capability 
assessment strategy. Initial efforts by DHS have produced pilot capability assessments, 
but how useful these pilots have been has not been fully examined. The purpose of this 
thesis is to explore how capability-assessment efforts may be improved by focusing on 
the process. Findings from the thesis will be used to make recommendations to inform 
capability assessment strategy development. 


16 Gail Thomas, Susan Hocevar and Erik Jansen, A Diagnostic Approach to Building Collaborative 
Capacity in an Interagency Context (Monterey, CA: Naval Postgraduate School, 2006), 2. 

17 United States Government Accountability Office, Results-Oriented Government: Practices that 
Enhance and Sustain Collaboration Among Federal Agencies, GAO-05-15 (Washington, DC: GAO, 2005), 
1 . 

18 United States Government Accountability Office, High Risk Series and Update, GAO-05-207 
(Washington, DC: GAO, 2005), 1. 
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II. ARGUMENTS 


The core claim of this thesis is that the process for performing capability 
assessments in homeland security must be studied to determine which methodology 
provides the most reasonable investment of resources, based on expected outcomes. The 
expected outcomes of the assessment are that it creates a snapshot of current capability 
status and causes additional benefits to be derived from the process of perfonning an 
assessment. 

The value of capability assessments in homeland security results from their 
foundational role in capability-based planning. Understanding the limits of capability 
assessment benefits local officials who make investment decisions based upon the results 
of an assessment. 

HSPD 8, the NPG, states that the nation can become prepared by developing 
appropriate threat-based capabilities. 19 The Goal further defines capabilities-based 
planning as the chosen mechanism for guiding Homeland Security evolution. The first 
step in capability-based planning is assessing what capabilities exist, which capabilities 
do not, and how to measure changes over time. 20 Focusing on the first step in the process 
will improve the effect of planning efforts at all stages of the process and is, therefore, the 
focus of this thesis. 21 

To satisfy the capability-based planning demands, an assessment process should 
be trusted by those charged with using the results. Trust in the assessment will allow a 
confidence that investments made to correct gaps identified by the assessment are 
defensible and proper use of taxpayer funds. This thesis will examine assessment 
strategies and, using data gained from a series of pilot assessments, recommend a strategy 
for conducting assessments that satisfies the needs of the homeland security community. 

19 George Bush, Homeland Security Presidential Directive/HSPD-8 (Washington, DC: The White 
House, December 2004), 2. 

20 Paul Davis, Analytic Architecture for Capability-based Planning, Mission System Analysis, and 
Transformation (Santa Monica, CA: RAND, 2002), xii. 

21 Ibid., xii. 
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In its short history, DHS has failed to develop a relationship of trust with local 
officials. When talking about the collaborative and information-sharing processes at 
DHS, Representative Bennie Thompson states, “The department must start querying state 
and local organizations on their existing initiatives and develop better collaborative and 
coordinating tools.” 22 The faults in this relationship are potential barriers to obtaining the 
necessary data from the assessment process. To ensure the assessment creates data that is 
systematic, repeatable, consistent and useful, it should be collected in a manner that 
recognizes a lack of trust by local government of DHS. To improve trust, an assessment 
process that creates collaboration by identifying local data needs are transparent, and 
shows the benefits of engaged participation, will allow the development of strategic 
planning processes to improve homeland security. One way this process will improve 
planning is to provide a database to measure improvement in preparedness over time. 
This process should also create an understanding of the interdependencies at local and 
regional levels required to achieve sufficient levels of preparedness. 

This thesis examines how collaboration in the assessment process produces 
outcomes, or impacts that improve homeland security preparedness. The discovery of 
capability gaps that otherwise would not have been discovered as a result of collaboration 
in the assessment process is an example of the benefit of collaboration. If collaboration 
does produce beneficial outcomes, then the starting point for understanding these benefits 
is the assessment process. 

One of the DHS goals in the assessment process is to create an understanding of 
preparedness activities at the local level. If collaboration produces benefits to 
preparedness, then collaboration should be included at initial stages of the planning 
efforts: the assessment process. Inclusion of collaborative effort at the earliest stages may 
create opportunities, through the assessment process, for future collaboration in support 
of an established national plan for preparedness. 


22 Bennie Thompson, “Department of Homeland Security churns out poorly developed schemes,” The 
Hill. June 5, 2007, http://thehill.com/leading-the-news/department-of-homeland-security-churns-out- 
poorly-developed-schemes-2007-06-05.html (accessed March 16, 2007). 


12 



An area of challenge is that DHS has created a problem by adopting capability- 
based planning as the strategy for securing the homeland. This argument is supported by 
the adoption of effects-based planning by the military and supported by the Office of 
Management and Budget. This challenge has a corollary argument: The planning efforts 
are too complex and the TCL is given too much weight in defining the outcomes of 
planning efforts. These challenges would argue that relying on state and local 
government to produce data — data that meets the stated goal of supporting strategic 
planning efforts and assessing preparedness — is impossible because of the variety of 
communities in our nation and the complexity of the TCL. 

The following comparative case study in a large urban area focuses on these 
controversies. For instance, the research suggests the extent to which data gathered in an 
assessment process is systematic, repeatable, consistent, and useful to a complex strategic 
planning process. These analyses also lead to a discussion of strategies to improve 
homeland security preparedness by improving the assessment process. Capability-based 
planning efforts, for example, may have more value if they are incorporated into an 
effects-based planning process that focuses more directly on how specific capabilities — 
even if they vary across jurisdictions — produce desired outcomes. 
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III. METHODOLOGY 


A. OVERVIEW 

The methodology employed in this thesis is a specific, localized test of the effect 
of altering a portion of the assessment process. The comparative case study is designed 
to examine whether and how an assessment process produces measures of capabilities 
and adds value to participants. The methodology employed in the study will include 
using a portion of the Self-Assessment Tool developed by a DHS Pilot Capability 
Working Group. This tool is designed to measure regional progress toward meeting the 
NPG. 23 Additionally, the working group expects that through the assessment process 
additional benefits for participants will be created. These benefits include opportunity for 
future collaboration in developing homeland security capabilities and an opportunity for 
participants to increase their knowledge of local capabilities. 

The study will utilize focus groups composed of existing multi-agency, multi- 
jurisdictional committees from the National Capital Region. These groups will employ 
different processes to conduct a self-assessment of regional capability. The processes to 
be employed are designed to create different opportunities for convenience and 
collaboration as assessment group members work toward completion of the self- 
assessment tool. 

Developing a methodology for understanding the effect of process, as it relates to 
opportunity for participation by subject-matter experts, is important to ensuring 
participation of subject-matter experts. Multiple demands on the schedules of the 
subject-matter experts limit opportunity for their participation in the self-assessment 
process. 24 Developing a process that allows for greater participation by homeland 
security professionals should cause more accurate and complete assessments to be 

22 Lauren Fernandez, Pilot Capabilities Assessment Orientation Brief, Pilot Capabilities Assessment 
Working Group Meeting (DHS: Washington, DC, February 22, 2006), slide 8. 

24 Lauren Fernandez, “Pilot Version 2.0: Minnesota Update and Discussion,” Pilot Capabilities 
Assessment Working Group Meeting (DH: Washington DC, June 27, 2006), slide 12. 
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conducted. Factors affecting opportunity to participate include the time needed to 
complete the assessment process and the convenience of scheduling the time 
commitment. 

The data collected in the study will be analyzed to compare reliability of results 
across different processes. Structured interviews with assessment participants will 
examine opinions for the utility of the assessment data and discover collaborative benefits 
produced by the assessment processes. 

B. ASSESSMENT PROCESSES 

In the study, three different assessment processes were employed to complete a 
self assessment. Each of the assessment processes utilized the same measures to rate 
status of the Mass Surge and Hazardous Materials and Weapons of Mass Destruction 
Response and Decontamination capabilities. Each of these processes is designed to 
provide varied opportunities for collaboration. 

Assessment Process 1 creates the most convenient opportunity for participation 
because the participant controls both the time and the location of the assessment. This 
assessment process creates the least opportunity for collaboration and discovering 
knowledge about the capability because the participants can choose to work alone to 
complete the assessment. In this process, assessment group members will be directed to 
complete the self assessment by a date two weeks in advance. The ratings of all 
participants will be aggregated into a final rating as to the status of the measured 
capability. The final output of Assessment Process 1 is a report on the capability status. 
This process will be employed in the assessment conducted by Assessment Group 1. 

Assessment Process 2 creates a mixed opportunity for convenience and 
collaboration. Step one of the process is to perform Assessment Process 1. Step two of 
the process is added after aggregation. Group members will be presented the aggregated 
data and provided the opportunity to build consensus to a final rating. The choices for 
the assessment group will be to accept the rating as accurate, or change the rating to 
reflect the actual status. This process increases the opportunity to collaborate on the 
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results of the assessment by initiating discussion on capability status to reach a consensus 
rating. Members of Assessment Group 2 will utilize this process to perform their 
assessment. 

Assessment Process 3 employes a process to complete the self assessment that 
limits opportunity for participation by requiring the assessment group member to dedicate 
a specific block of time at a specific location to complete the process. The benefit to this 
process is that the design increases collaborative opportunity for participants by causing 
them to work together to reach a consensus assessment rating for each measure. 
Assessment Group 3 and 4 will use this process. 

C. ASSESSMENT GROUPS 

Evaluating how process affects the outcomes requires that similar assessment 
groups and a similar assessment tools are employed in different assessment processes. 
This research will use existing regional, multidiscipline committees to conduct the self 
assessment utilizing different processes. 

1. Assessment Group 1 

The National Capitol Region, Mass Decontamination Working Group is 
comprised of representatives of fire departments, emergency medical services, police 
departments, health departments, hospitals, hazardous materials teams, emergency 
management, technology offices, and transportation offices from state and local 
governments. This mission of this group is to develop and implement policies and plans 
to improve Mass Decontamination capabilities within the National Capital Region. 

Members of this group are decision makers from their respective agencies who 
have been tasked with developing innovative solutions to complex problems. It is the 
expectation of regional leaders that members appointed to this committee have intimate 
knowledge of Homeland Security issues in the National Capital Region. These members 
should also be able to understand how issues of one discipline or jurisdiction affect other 
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disciplines or jurisdictions. Many of the group members represent their agencies in the 
Homeland Security Grant Process and are leaders of other related committees and 
organizations. 

Assessment Group 1 will be employed to examine a process that eliminates the 
need for members to work together when completing the assessment. Time commitment 
is a problem for senior level managers of any type of organization. Development of a 
method that reduces time impact by increasing flexibility may increase participation in 
the assessment process and improve the accuracy of results by encouraging the 
participation of the target audience. 

To examine the reduced time processes, group members will complete the 
assessment tool individually. When completed, the assessment tool will be returned to a 
coordinator who will compile the results of the assessment. The result will be the 
average rating to each measure by the individual group members. In this process, the 
members work by themselves at a time and place of their choosing. 

2. Assessment Group 2 

The District of Columbia Strategic Planning Committee, Detection Research and 
Development Subcommittee is tasked with evaluating and integrating technologies for 
chemical, biological, radiation, nuclear and explosives detection with everyday polices as 
defined in the District of Columbia Strategic Plan. This committee must integrate needs 
of the district government with those of federal, state and private sector partners in order 
to provide layered detection capabilities within the National Capitol Region. 

Members of this committee are subject-matter experts in detection technologies or 
response policies of their respective agencies. The members routinely collaborate with 
partners outside of their home agencies in furtherance of the goals and objectives of the 
committee. 

Group two members will be employed in a process that studies the results of 
members completing the assessment on their own, then coming together to reach 
consensus on the results. In theory, this method will shorten the time needed to work 
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together while still allowing members collaborative opportunity by virtue of the 
consensus process. This method may reduce the time impact of the assessment process 
by increasing flexibility in when and how the survey is completed. This method is 
similar to the read ahead method used in meeting environments to assure that members 
are familiar with material to be covered and, as a result, reduce the amount of time spent 
in a meeting. 


3. Assessment Group 3 

The Metropolitan Washington Council of Governments, Fire Chiefs Hazardous 
Materials Subcommittee mission is to coordinate the policies and procedures for 
hazardous materials response teams in the National Capitol Region. This committee has 
been in existence since 1982 and has developed and implemented regional policies for 
Radiation Response, Rapid Response and overseen regional training programs in 
Weapons of Mass Destruction Training for First Responders. Federal, state and local 
governments are represented on the committee as are Department of Defense assets, the 
private sector, our neighboring Urban Area Security Initiative region and many 
disciplines. 

Group 3 will be used to examine the consensus method for completing the 
assessment. In this process, the members of the Hazardous Materials Subcommittee will 
convene with the purpose of completing the assessment tool. For each measure, the 
members will come to consensus on the status of the measure. This result will be 
recorded as the status of this capability. 

Consensus is the method suggested by Pilot Capabilities Assessment Project 
Working Group as preferred to complete the assessment tool. Consensus is the preferred 
method because it is believed the process will foster collaborative relationships that may 
extend beyond completion of the assessment tool. The working group believes that the 
collaborative regional relationships are important to building improved capability with 
the NPG. 25 

25 Department of Homeland Security, Pilot Capability Assessment Plan Working Document (DHS: 
Washington, DC, March 3, 2006), 11 
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4. 


Assessment Group 4 


This study was originally designed with three groups. During the assessment 
process with Assessment Group 1, a parallel assessment group was initiated by group 
members. Because the research question involves identifying both the benefits of 
collaborative assessment process and future collaborative opportunities arising from the 
assessment process, I decided to include this newly formed group as Assessment Group 
4. The details of how this group was formed are discussed in Chapter IV, Analysis, 
Conclusion. 

This group is composed of members of the Council of Governments Medical 
Surge Hospitals and Health Subcommittee. This group represents local public health 
officials, private sector representatives, emergency managers, and first responders. In the 
case study, this group employed the consensus method for completing the assessment. In 
this process, the members of the group convened with the purpose of completing the 
assessment tool. For each measure, the members came to consensus on the status of the 
measure. Their results were recorded as the status of this capability. 

D. INTERVIEWS 

The product of each group — a completed capability assessment perfonned with 
the Self Assessment Tool — provides the data used in this analysis. The first step in the 
analysis is to discover if the assessment tool can be used to facilitate data measurements 
that are reflective of the capabilities being measured. To reach this goal, results of the 
group products will be compared to each other in an attempt to discover similarities and 
differences. This comparison of data sets will be used to identify any effect of the 
different processes in performing the assessment. 

Interviews with focus group members will be structured and include questions 
pertaining to how the process affected how the assessment data would be used and, 
outside of the process, if there are other factors that inhibit participation in the assessment 
process. A comparison of the results of the assessments and the results of the interviews 
will provide the data for this thesis. 
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From this analysis, questions for each of the participants were developed. The 
results of the interviews were used to confirm observations about collaboration and 
inform policy recommendations about which of the assessment processes holds the most 
promise for providing a reliable capability assessment and creating additional benefits. 

E. LIMITS OF THE EMPLOYED METHODOLOGY 

There are limits to how the results of these case studies should be compared and 
interpreted. The methodology is designed to examine the process used in the assessment. 
This study should have been performed using the multiple randomly-assigned assessment 
groups to perform the assessment utilizing a different process. This was not done. 
Instead, the assessment groups were composed of existing groups. The decision to 
perform the case studies in this manner was made because the DHS proposed 
methodology performs the assessment one time, by assessment groups with similar 
compositions to the study assessment groups. 

The use of existing groups as assessment teams also serves to limit how data from 
this study can be interpreted. A more reliable method would be to utilize randomly 
selected subject-matter experts to perform the assessments. In reality, the small size of 
the subject-matter expert community, and the heavy time demands placed upon them 
because of their expertise, eliminated this option. This decision is also consistent with 
how the region has dealt with performing capabilities assessments in the past, assigning 
the assessment responsibility to an existing group. 

The use of existing groups does not control for selection bias, and as a result some 
of the suggest findings may have alternative explanations. The alternative explanations 
may come from group composition and not the change in process. This limit will be 
especially important to consider when comparing group results. 

These methodology compromises ultimately limit the size and scope of the study. 
As a result, the data produced by the case studies should be recognized for its 
weaknesses, and the analysis of the data collected during the study can only be 
recognized as suggestive or exploratory findings. 
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IV. ANALYSIS 


A. ANALYSIS INTRODUCTION 

This thesis examined a proposed methodology for perfonning the National 
Capabilities Assessment required by HSPD 8. Within the proposed methodology, a 
question of which process would be most appropriate to detennine the status of 
capabilities became pertinent. The study within this thesis was developed to examine 
potential assessment processes to determine if a proposed approach had benefits over 
other possible approaches. 

This analysis is limited to the small set of capabilities surveyed in the comparative 
case study. The study is performed within the guidelines of the proposed methodology. 
The study produced a data set that is suggestive of results that may be found in other 
Urban Area Security Initiative regions, but cannot claim to represent all of homeland 
security. The data set represents many of the homeland security disciplines, but does not 
provide multiple viewpoints from some of the support disciplines. Interviews with survey 
participants were used to clarify the existence of trends in the data and confirm 
observations about collaboration, validity and reliability. 

B. CAPABILITY ASSESSMENT COMPARATIVE CASE STUDY 

The Pilot Capability Assessment Process is designed to employ a group of 
subject-matter experts to assess the status of the various capabilities defined in the TCL. 
The assessment process is guided through the use a self-assessment tool. This tool 
employs a set of measures for each capability. Using the scoring scale in Figure 1, the 
measures are rated by the assessment group and collectively represent the status of the 
capability. 
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Figure 1. 


Pilot Capability Self Assessment Tool Scoring Scale. 26 


Label 

Explanation 

Scale Value 

No Progress 

A score of 0 indicates that, while this aspect of [name the 
capability] is applicable to the region, no progress has 
been made towards achieving the identified objective. This 
may be because there has been no activity in this area, or 
because insurmountable barriers exist. 

0 

Limited 

Progress 

Regions should score themselves at the lower to middle 
part of this range if preliminary efforts have been initiated. 
Needs related to this objective have been recognized, and 
the region is just beginning to identify requirements in this 
area. Few if any steps have been taken so far. 

Regions should score themselves at the middle to upper 
part of this range if the region has analyzed needs and 
understands requirements, and has taken specific steps 
toward achieving the objective. These steps may include 
initial plans to develop this aspect of [name the capability], 
allocation of resources, and identification personnel 
responsible for achievement of the objective. 

1 

2 

3 

Moderate 

Progress 

Regions should score themselves at the lower to middle 
part of this range if significant efforts are under way but the 
objective has not yet been fulfilled. Important gaps remain 
and/or challenges that could potentially undermine 
achievement exist and have not yet been resolved. 

Regions should score themselves at the middle to upper 
part of this range if significant efforts are under way and 
specific examples of progress in this area can be identified. 
Strategies for closing gaps and overcoming barriers to 
success are being developed and initiated. 

4 

5 

6 

Substantial 

Progress 

Regions should score themselves at the lower to middle 
part of this range if efforts to achieve this objective are 
established and stable. Some weaknesses or barriers that 
prevent success persist, but strategies to resolve them are 
documented and well under way. 

Regions should score themselves at the middle to upper 
part of this range if efforts in this area are mature. Few 
gaps or barriers to success remain. None are significant. 
Evidence documenting this level of progress is readily 
available. Evidence may include After Action Reports from 
exercises or events where this aspect of capability was 
demonstrated. 

7 

8 

9 

Objective 

Achieved 

A score of 10 indicates that the region has fully achieved 
this objective with regard to [name the capability]. All 
barriers to success have been overcome. Strengths are 
robust and likely to be sustained. Evidence is readily 
available attesting to this level of achievement. 

10 

Not Applicable 

The aspect of [name the capability] that this question 
describes is not applicable to the region. 

N/A 


26 Self Assessment Tool Version 2.0, Pilot Assessment Working Group (DHS, FEMS, NPD: 
Washington, DC: January 29, 2007), Tear Out Sheet Tab. 
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The National Capital Region employs a homeland security strategy based upon 
the National Response Plan. 27 Under this strategy, the Target Capabilities in the NPG are 
assigned to Emergency Support Functions for implementation. The two Target 
Capabilities for assessment were selected to assure that data came from a cross-section of 
emergency support functions and from target capabilities with different emergency 
support function responsibility. 

Health is the designated lead for Emergency Support Function # 8, Health, Mental 
Health and Medical Services. This group has been tasked to lead the implementation of 
the Mass Surge capabilities. The effort is supported by the emergency support functions 
for # 1, Transportation; # 4, Firefighting, # 5, Communications and Planning; # 13, Law 
Enforcement; and #15, Donations and Volunteer Management. 

The Fire Department has been designated as the lead agency for Emergency 
Support Function # 10, Hazardous Materials. This group has been assigned the 
responsibility to implement the Hazardous Materials/WMD Response and 
Decontamination Target Capability. This effort is supported by # 1, Transportation; # 5, 
Communications and Planning; # 8, Health, Mental Health, and Medical Services; # 13, 
Law Enforcement; and #15, Donations and Volunteer Management. 

The selection of these groups assures a similar support base across capabilities to 
be measured and will provide continuity in the assessment process. The members 
participating in the assessment are people who work together on many of the same issues 
being assessed as a regular course of their employment. 

1. Assessment Tool 

A Pilot Capability Assessment Working Group created the Self Assessment Tool 
to assess capabilities. This goal of this tool is to allow local preparedness officials to 
participate in a process of “capabilities assessment [that] produces valuable and relevant 
preparedness infonnation [and] supports a regional preparedness approach to planning 


27 The National Capital Response Plan is formulated but not ratified, but working drafts are in place. 
Planning and Response functions are routinely using the draft form of the plan. 
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and investment efforts.” 28 This study was not designed to examine the tool directly, but 
will examine potential inconsistencies in the tool or measures suggested by the data. 

The tool uses a series of measures that are designed to assess the progress of a 
region in implementation of the NPG. Selecting portions of the tool that provide the 
opportunity to measure the response of multiple disciplines across two dissimilar 
capabilities allows for an evaluation of the tool. For this survey, the Target Capabilities 
to be assessed are Mass Surge and Hazardous Materials/WMD Response and 
Decontamination. 

The tool is designed for a small group of subject-matter experts to evaluate a set 
of cross-capability measures about collaboration and a set of specific measures about a 
defined capability. The assessment process is guided through the use a Self Assessment 
Tool. This tool employs a set of measures for each capability. Using the scale in Figure 
1, the measures are rated by the assessment group and collectively represent the status of 
the capability. 

The scoring scale is comprised of a subjective numeric scale tied to labels and 
explanations of progress that are used to represent progress toward meeting a specified 
measure. The number scale is anchored on either end by the number “0,” indicating no 
progress toward meeting the measure and the number “10,” indicating the measure is 
completely satisfied. The number scale is further divided by explanations that create sets 
of three numbers into three labeled categories of effort. 

To develop the tool, the Pilot Capabilities Working Group refined characteristics 
of a good assessment tool for data collection using after-action reports on pilot 
assessment programs conducted in Florida and Colorado. They declared that survey 
measures should have the following characteristics: 

• The measures should be a single-stage question, not a multipart question. 

• The measures should provide enough detail and information so the 
measure has universal meaning to respondents. 

28 Seamus Mooney, “Pilot Capabilities Assessment Update & Methodology Discussion,” Pilot 
Capability Assessment Working Group (FEMA, NPD: Washington, DC, August 15, 2007), 4. 
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• Training and resources measures should be separate from plans and 
equipment. 

• Equipment measures should address all aspects (procurement, training and 
maintenance) of the use of the equipment. 29 

2. Precision 

Because the label scale is grouped by threes, it can be inferred that the accepted 
level of precision through use of the tool is a range of three points. Therefore, any 
measures of progress by assessment groups that are within three points of each other 
constitute agreement on the rating given to a measure. 

The data in this analysis was obtained from an experiment that employed three 
different processes to complete a self assessment. Each of the self assessments utilize the 
same measures to rate the status of the Mass Surge and Hazardous Materials and 
Weapons of Mass Destruction Response and Decontamination capabilities. The 
assessments were performed by local subject-matter experts; the same people who have 
performed in assessment roles in the past and have responsibilities for implementing 
programs in the capability areas being assessed. 

The success of this comparative case study and the capability assessment assumes 
that the assembled subject-matter experts have the requisite knowledge needed to assess 
the measures, and that collaboration among these subject-matter experts will produce a 
valid and reliable convergence of ratings. 


29 Department of Homeland Security, Pilot Capabilities Assessment Project, Meeting Notes, October 
25, 2006 (Department of Homeland Security, Office of Grants and Training): 2. 
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Reliable 
Not Valid 


Valid 

Not Reliable 


Neither Reliable Both Reliable 
Nor Valid And Valid 


Figure 2. Validity and Reliability Examples. 30 


Figure 2 represents the possible outcome scenarios from our assessment groups. 
The red dot in the center represents the undetermined status of the measure, and the blue 
dots represent assessments of the measure. In this assessment process, a reliable and 
valid outcome is multiple subject-matter experts providing similar ratings for a measure. 

C. RELIABILITY 

In the comparative case study, four separate groups of subject-matter experts 
completed the assessment utilizing the measures in the Self Assessment Tool to rate the 
progress toward implementation of the Hazardous Materials / WMD Response and 
Decontamination Capability. Reliable data indicating capability status from this survey 
would be indicated by measures with scores grouped around a range of three points on 
the scale; or grouped together as indicated in Figure 2, “Reliable and Valid.” The range 
of three points was earlier described as the precision of the Pilot Self Assessment Tool. If 
the range exceeded three points across a specific measure, we could infer that the results 
for this assessment process are unreliable. Because this thesis is examining the effect of 
process when using the tool, this finding would suggest further study of the process and 
the assessment groups. 


30 William M. K. Trochim, “Reliability & Validity,” Research Methods Knowledge Base” (Web 
Center for Social Research Methods), October 10, 2006, 
http://wwsocialresearchmethods.net/kb/relandval.php , assessed July 17, 2007. 
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Figure 2 also shows an example; “Reliable, not Valid,” that can occur when the 
range of ratings is within a three-point range but does not adequately relate to the 
underlying capability status. Participants may agree, but in this example they may be 
agreeing on the basis of wrong information about the capability. An examination of any 
grouping warrants examination to determine validity. 



— Group 1 —■— Group 2 Group 3 Group 4 


Figure 3. Self Assessment Group Ratings, Hazardous Materials / WMD Response and 

Decontamination Capability. 


The chart in Figure 3 represents a comparison of the Self Assessment Group 
ratings for the fifteen measures in the Hazardous Materials / WMD Response and 
Decontamination capability assessment. This chart clearly shows that only measures 3, 
7, 10 and 15 produce a grouping of results within the desired three-point range for 
reliability. The remaining assessment results exceed the three-point range. This 
observation suggests the need for further study of the effect of the assessment process on 
the validity of the results. 
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Figure 4. Self Assessment Group Ratings, Mass Surge Capability. 


The chart in Figure 4 represents Self Assessment Group ratings for measures in 
the Mass Surge capability assessment. In this case, only measures 11 and 12 exceed the 
three-point range for accuracy. The finding from this comparison suggests that the 
different assessment processes produced reliable groupings of data. Further examination 
of this data will be completed to examine validity. 

An examination of the reliability of the assessment processes, as measured by 
group outputs, suggests that there could be problems in the Hazardous Materials / WMD 
Response and Decontamination assessment and, to a lesser degree, in the Mass Surge 
assessment. A finding that the assessment process failed to produce reliable results 
suggests the need for further examination of the methodology. If true, unreliability is an 
important finding about the capability assessment process. 

The findings in this section of the analysis explore problems with data reliability. 

These findings also suggest a need to examine the validity of the data collected. The 
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examination of validity will be prefaced by a discussion on collaboration, because the 
extent and methods of collaboration provides evidence on the validity of the assessment 
results. 

D. COLLABORATION AND CONVERGENCE 

In addition to providing data about capability status, the expected outcomes of the 
assessment process are that it will both educate and encourage collaboration. 31 This study 
provided an opportunity to explore the educational and collaborative aspects of the 
assessment process by examining the data and interviewing participants to understand 
how members came to agree on ratings for measures. The expected result from the study 
is that members of the assessment groups would come to the process with a certain level 
of understanding of the capability status. Also, as a result of collaboration with other 
group members, they would gain additional knowledge about the capability status, thus 
converging on the actual capability status. 

An observation of Assessment Group 2 allows an understanding of how 
individual assessment group members came to a consensus for group ratings. These 
observations show how the assessment process produced a final rating. This final rating 
should represent a convergence of the individual group member’s knowledge about the 
capability status. If a convergence in the ratings does not occur, problems related to 
group think and assessment design need to be considered. 

Observations of Assessment Group 2 were used to examine the collaborative 
effort employed to meet the goal of performing a multi-disciplined assessment. 
Interviews with individual group members were used to detennine if convergence 
occurred in reaching the final rating, or if agreement was influenced by other behaviors. 


31 Department of Homeland Security, "Pilot Capability’ Assessment Plan Working Document” (DHS: 
Washington, DC, March 3, 2006), 2. 
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Measure12 Final Determined Capability Status 


Figure 5. Assessment Group 2, Haz Mat / WMD Response and Decontamination, 
Measure 12, Individual Assessment versus Group Assessment. 


Figure 5 demonstrates a comparison of the level of progress ratings from the 
members of Assessment Group 2 for Measure 12; 

Our region is able to ensure the safety and security at the scene and 
support facilities it intends to use for WMD/Hazardous Materials 
Response and Decontamination. (That is, can your region determine and 
ensure the structural integrity, capability, and physical security of its 
HazMat Management facilities?) 32 

The chart demonstrates diversity of individual opinion about the status of 
Measure 12. The ratings range of eight points exceeds the three-point precision of the 
tool and would cause the assessment result to be considered unreliable if those views 
remained as the final opinion of the assessment group. The chart also demonstrates a 


32 Self Assessment Tool Version 2.0, Pilot Assessment Working Group (DHS, FEMS, NPD: 
Washington, DC: Hazardous Materials / WMD Response and Decontamination Tab, January 29, 2007. 
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comparison between individual determination of level of progress and final group 
determination of level of progress for the measure. Finally, the chart also shows where 
a member of the health discipline did not provide a rating for the measure because of 
uncertainty as to the level of progress. 33 



Figure 6. Assessment Group 2, Haz Mat / WMD Response and Decontamination, 
Measure 6, Individual Assessment versus Group Assessment. 


Figure 6 demonstrates a comparison of level of progress ratings from members of 
Assessment Group 2 for Measure 6: 


33 Instructions to assessment team members completing the survey were to complete it to complete the 
survey to the best of their ability, follow up questions from members resulted in assessment team members 
being directed to not rank measures they did not “know the answer to.” 
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Our region’s plans for WMD/Hazardous Materials Response and 
Decontamination address substance identification equipment (e.g., bases, 
vapors, liquids, solids, biologicals like white powder). 34 

The chart demonstrates diversity of individual opinion about the status of 
Measure 6. The ratings range of five points exceeds the three-point precision of the tool 
and would cause the assessment result to be considered unreliable if those views 
remained as the final opinion of the assessment group. The chart also demonstrates a 
comparison between individual determination of level of progress and final group 
determination of level of progress for the measure. Finally, the chart also shows where 
a member of the health disciplines did not provide a rating for the measure because of 
uncertainty as to the level of progress. 



Measure14 —I— Final Determined Capability Staus 


Figure 7. Assessment Group 2, Mass Surge, Measure 14, Individual Assessment versus 

Group Assessment. 


34 Self Assessment Tool Version 2.0, Pilot Assessment Working Group (DHS, FEMS, NPD: 
Washington, DC: Hazardous Materials / WMD Response and Decontamination Tab, January 29, 2007. 


34 




















Figure 7 demonstrates a comparison of the level of progress ratings from the 
members of Assessment Group 2 for Measure 14: 

Our region maintains databases that track the status of resources (e.g., 

medications, medical professionals) available to support Medical Surge. 35 

The chart demonstrates diversity of individual opinion about status of Measure 
14. The ratings range of six points exceeds the three-point precision of the tool and would 
cause the assessment rating to be considered unreliable if those views remained as the 
final opinion of the assessment Group. The chart also demonstrates a comparison 
between individual detennination of level of progress and final group determination of 
level of progress for the measure. Finally, the chart also shows where members of the 
police and fire disciplines did not rate the measure because of uncertainty about the level 
of progress. 

E. VALIDITY 

The foundation of this assessment methodology rests on the ability of subject- 
matter experts to reach an accurate conclusion on the status of a capability. In the 
proposed methodology, the subject-matter experts are selected by state and local 
governments. Within local and state government, the functional area being measured is a 
niche; at this time much of the detailed knowledge of homeland security activities is with 
a small cadre of personnel. Because the community is small, care must be taken to 
ensure that assessment groups are representative of a cross-section of this community in 
order to ensure diversity of opinion. Ultimately, in this study, validity is measured by 
agreement of the groups and members as measured in assessment ratings. 

The charts presented represent the wide range of opinions on the level of progress 
being made towards the capabilities held by individual members of the assessment group, 
including examples where members felt they did not know the level of progress for a 
measure. If the average rating reflects the member’s “informed” opinion about the status 

35 Self Assessment Tool Version 2.0, Pilot Assessment Working Group (DHS, FEMS, NPD: 
Washington, DC: Mass Surge Tab, January 29, 2007. 
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of the capability, then it is necessary to understand how the assessment groups reached 
consensus on the rating for a measure. This point is critical to understanding the validity 
of each measure and, through each measure, the entire process. The design of the tool 
expects consensus to create validity as assessment group members employ their expertise 
to validate the assessment results. 

1. Validity Assessment Group 2 

When Assessment Group 2 met to reach consensus on the assigned rating for each 
measure, group members accepted the aggregated result each time. In the consensus 
process, there were discussions about the level of progress, and pointed questions asked 
between disciplines. These discussions, however failed to move the group from the 
aggregated rating. The questions asked between members of the group were focused on 
learning about the status of the capability, as opposed to efforts to have other members 
change their opinion about a capability status. 

For example, members of the police discipline inquired with the health discipline 
about integration of security expectations in the measures. These discussions exhibited 
the characteristics of free flow of information and discovery that are the elements of 
collaborative effort. 36 

Interviews were utilized to examine the occurrence of convergence in the 
assessment ratings, specifically to examine why members agreed to accept ratings that 
differed greatly from their own. Interviews with group members found that members 
routinely accepted the average rating because they were uncertain of their original rating. 
Evidence of the lack of confidence in the original rating or the final rating is 
demonstrated in the interview process. Reasons given for accepting the aggregation 
included: 


36 Hocevar, Susan, Gail Thomas and Erik Jansen, “Building Collaborative Capacity: An Innovative 
Strategy for Homeland Security Preparedness,” Innovation Through Collaboration, Advances in 
Interdisciplinary Studies of Work Teams 12 (Elseier Ltd: 2005), 264. 
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• Looking at the scores now, I went too high. 37 

• I gave an honest answer, not sure everyone else gave honest answer. 38 

• I am an optimist, but the group might be producing a more accurate rating. 39 

It is a significant finding that group members accepted the mean ratings without a 
single change every time. They accepted the changes despite evidence of a wide range of 
opinions as to what the correct rating could have been. This finding is not an example of 
convergence in the sense that convergence equals knowledge gained. It is a finding that 
members changed their assessments because they lacked confidence in their knowledge. 
This finding is true for both capabilities assessed. This finding suggests exploring the 
assumption that subject-matter experts have the expertise to conduct reliable and valid 
capability assessments. 

2. Validity of Assessment Group 1 

To support the suggestion of validity problems from the analysis of data from 
Assessment Group 2, an examination of the data collected from Assessment Group 1 was 
conducted. This data is important because it also suggests that the selected subject-matter 
experts lack the knowledge to assess capabilities against the proposed measures. 

Figure 8 demonstrates individual ratings from Assessment Group 1 for Hazardous 
Materials / WMD Response and Decontamination, Measure 15, “To what extent has your 
region exercised its ability to implement WMD/Hazardous Materials Response and 
Decontamination procedures (e.g., exercised response to pre-identified hazards, 
decontaminated victims).” 40 The chart demonstrates diversity of individual opinion 
about the status of Measure 15. The eight-point ratings range exceeds the three-point 
precision of the tool. The chart also demonstrates a comparison between individual 

37 Interview of Participant 5 by author, Washington, DC, June 6, 2007. 

38 Interview of Participant 6 by author, Washington, DC, June 5, 2007. 

39 Interview of Participant 7 by author, Washington, DC, June 5, 2007. 

40 Self Assessment Tool Version 2.0, Pilot Assessment Working Group (DHS, FEMS, NPD: 
Washington, DC: Hazardous Materials / WMD Response and Decontamination Tab, January 29, 2007. 
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determination of level of progress and final group determination of level of progress. 
Finally, the chart also shows where a member of the fire discipline did not rate the 
measure because of uncertainty as to the level of progress. 



Measure15 Final Assessed Rating"] 


Figure 8. Assessment Group 1, Hazardous Materials / WMD Response and 

Decontamination, Measure 15, Individual Assessment versus Group Assessment. 


Figure 9 demonstrates individual ratings from Assessment Group 1 for Mass 
Surge, Measure 3: 

Our region’s plans address the use of existing facilities in support of 
Medical Surge (e.g., hospitals, clinics, extended care facilities). 41 

The chart demonstrates diversity of individual opinion as to level of progress 
towards Measure 3. The seven-point rating range exceeds the three-point precision of the 
tool. The chart also demonstrates a comparison between individual determination of 

41 Self Assessment Tool Version 2.0, Pilot Assessment Working Group, (DHS, FEMS, NPD: 
Washington, DC: Mass Surge Tab January 29, 2007. 
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level of progress and final group determination of level of progress. Finally, the chart 
also shows where two members of the emergency management discipline did not rate the 
measure because of uncertainty as to the level of progress. 



Measure3 


- Final Determined Capability Status | 


Figure 9. Assessment Group 1, Mass Surge, Measure 3, Individual Assessment versus 

Group Assessment. 


At the conclusion of the comparative case study, members of Assessment Group 1 
were individually presented with the aggregated results and queried about the final 
ratings, the relationship of the final rating to their rating and if they would feel 
comfortable using the results of the assessment in planning for future efforts. 

Assessment Group 1 members reacted in a similar manner to Assessment Group 2 
members in accepting the final aggregated ratings as more accurate than their own 
opinion on the status of the capability. Many of the members felt they might use this data 
in future planning efforts. This finding is troublesome, because it presents a dilemma of 
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inaccurate information about the status of a capability being interjected into the 
preparedness cycle and capability-based planning efforts required by HSPD 8. 

The data suggests that disagreement in the assessment group members’ 
confidence in their individual knowledge was exhibited on multiple occasions. One of 
the police officials who completed the survey felt that she would use the data to 
determine funding or project priorities between the assessed capabilities. She did say that 
she was unsure about other uses of the assessment results. 42 A member of the emergency 
management discipline, who did not provide ratings for the Mass Surge Capability, felt 
that the results would need further study to determine if they held value. This member 
went on to say that the assessment results were accurate for Mass Surge, based on his 
limited knowledge, and he was confident in the assessment results for Hazardous 
Materials / WMD Response and Decontamination. 

3. Validity Conclusions 

The validity of this assessment methodology is based on the subject-matter 
experts’ ability to reach an accurate conclusion about the status of a capability. As 
described earlier, the community of experts is a small group with wide-ranging 
responsibilities and needs. In this small, diverse group, it is important the assessment 
process represent all members. Representing all members will ensure the diversity of 
opinion necessary to thoroughly examine and accurately assess the status of a capability. 

There is a limiting factor in this discussion about validity, in that the context of 
this thesis validity can only refer to the agreement among the assessment teams. Because 
of the aforementioned limits of the study methodology, the findings presented in this 
section are not definitive, but explore a potential problem in the proposed assessment 
process. 

The four groups employed in this study utilizing the Self Assessment Tool are 
representative of the homeland security community. Members in this study represent 
fourteen different disciplines, federal, state, local government and the private sector. 

42 Interview Participant 3 by author, Washington DC, June 5, 2007. 
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More importantly, the assessment group members are leaders of projects or committees 
implementing the capabilities being measured. These members and the groups they 
represent are the right people to perform a macro level capability assessment of this 
specialized knowledge area because they have the requisite specialized knowledge. 

Assessment Group 1 was composed of members of an existing committee that is 
working to implement regional mass decontamination standards. Assessment Group 2 
was composed of an existing committee that meets to discuss detection technology and 
implementation programs, plans and purchases in compliance with the goals of the 
District of Columbia Response Plan. Assessment Group 3 was composed of members of 
a regional hazardous materials response committee. Assessment Group 4 was composed 
of a regional hospital-centered health care committee. 

These committees and groups are responsible for updating response plans, 
submitting grant applications and implementing plans to raise their community’s level of 
preparedness. These people are the experts in their fields. The finding — that these 
people lacked confidence in the measures — points to a number of potential problems. 
The problems could be their knowledge, the measures, or the methodology. What is 
suggested by this study is that the capability assessment methodology and the subject- 
matter experts are misaligned. The result of this misalignment is the performance of an 
inaccurate capability assessment. 

An earlier suggestion — that the Mass Surge assessment process provided reliable 
results — may be questioned by an occurrence during the Mass Surge capability 
assessment. An examination of Figure 8 shows that data from Assessment Group 3 is not 
included. This data is not included because Assessment Group 3 declined to complete 
the Mass Surge capability assessment. Members of Group 3 asserted they did not have 
the requisite knowledge to accurately complete the assessment. They felt it would be 
better to abstain from the process than to provide inaccurate results. 43 This suggests a 
significant problem related to validity. The suggestion is that subject-matter experts were 
not subject-matter experts for the purposes of the capability assessment process. 

43 Minutes of the Metropolitan Washington Council of Governments, Fire Chief s Hazardous 
Materials Subcommittee Meeting, complied by Sean Brooks, April 13, 2007. 
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Non-response to a question or category of questions was an unexpected 
occurrence in this study. The findings here suggest a pattern that questions the basis for 
the capability assessment methodology. Subject-matter experts with oversight and 
planning responsibilities responded to the survey or the assessment process by 
apologizing for not having the “correct” answers or explaining that they did not have the 
requisite knowledge of regional efforts to rate regional progress toward the goal. 44 One 
fire chief apologized for not collaborating to obtain more accurate infonnation to 
complete the survey; he also felt that he should know more about the answers to the Mass 
Surge capability. 45 This admitted lack of knowledge is consistent with the wide range of 
individual ratings given to measures. 

Analysis of the data collected from the assessment process suggests further 
exploration of validity. The suggested findings related to validity are centered on the 
assumption of the expertise of the subject-matter experts selected to perfonn the 
assessments. The problem was that some members admitted to lacking the knowledge 
they were expected to have to complete the survey. 

These suggested findings create a problem with data collected. In the assessment 
process, the status of the capability is unknown; if the assessment ratings are determined 
to be invalid, then there is no way to determine the status of the capability. The next 
section, on collaboration and convergence, will be used to examine knowledge levels of 
the subject-matter experts as it relates to this process. 

F. ASSESSMENT DATA BY DISCIPLINE 

When presented with the results of the survey, members of one discipline 
inevitably wanted to know how the other disciplines rated a measure in comparison to the 
rating of their discipline. This comparative case study provided some data in that area 
and suggests some trends that should be considered in future methodology development. 


44 Participant 1 email to author April 3, 2007, Participant 2 email to author, March 27, 2007, and 
Interview of Participant 3 by author, June 5, 2007. 

45 Participant 4 email to author, March 30, 2007. 
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Figure 10. Mass Surge Assessment Ratings by Disciplines/Group. 


In the Mass Surge Capability, the health discipline is designated the lead 
Emergency Support Function for the development and implementation of plans and 
programs to ensure that adequate Mass Surge capability is available. In Figure 11, the 
chart presents the assessment given to each measure according to discipline. This chart 
shows that members of the health discipline tended to provide higher ratings for this 
capability than all of the other disciplines except police. The trending shows the police 
and health discipline generally agreed on the ratings for performance measures. 

Figure 10 also demonstrates an interesting finding about group versus individual 
ratings. The health discipline’s aggregated scores are, on average, far above Assessment 
Group 3’s average. This is significant because it demonstrates the possible differences 
between an aggregated scoring and a consensus scoring methodology. Given the 
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conflicted data presented in this thesis, it is impossible to determine which methodology 
provided a more accurate assessment of capabilities. 

The emergency management discipline is a bellwether in this survey. This 
discipline has overall coordination and planning responsibilities that cross all disciplines 
and capabilities. An argument can be made that if the higher assessment ratings assigned 
by the health discipline were accurate the emergency management discipline would also 
have provided a rating above average, instead of the consistently lower rating given. 

The data set used here is inconclusive but suggests that coordination portions of 
the Mass Surge capability plan are incomplete. Three members of the emergency 
management discipline participated in the individual portions of the survey. Two 
members declined to complete the Mass Surge Self Assessment of the survey due to their 
lack of subject-matter expertise in the area; leaving the discipline rating reflecting the 
views of a single person. A single person assigning low ratings is not definitive, but 
those results, accompanied by the abstentions of two members, point to a lack of 
knowledge of the Mass Surge capability plans among members of the agency responsible 
for coordination among agencies. 

The finding — that the members of a discipline with high levels of responsibility 
within a capability assign higher scores to that capability than other disciplines — was 
also shown in the assessment of the Hazardous Materials/WMD Response and 
Decontamination capability. When broken down by discipline, the data showed the 
police disciplines assigned higher rating than the average rating of all disciplines for each 
measure. The fire discipline assigned a higher-than-average rating in ten of fourteen 
measures. 
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Figure 11. Hazardous Materials/WMD Assessment rating by discipline 


Figure 11 reinforces a finding from the Mass Surge capability assessment. 
Emergency Management Agency had difficulty in assessing the Mass Surge capability; in 
the Hazardous Materials/WMD Response and Decontamination capability the data shows 
that the emergency management personnel routinely gave lower ratings than other 
disciplines. Figure 11 shows the measures in which the emergency management 
personnel assigned low ratings that exceeded the standard deviation for all assessors. 

In regional response plans, the fire discipline is assigned responsibility for 
implementing plans and programs to improve the Hazardous Materials/WMD Response 
and Decontamination capability. The fire discipline was found to provide higher than 
average mean performance ratings. This should be expected, as the group is most 
familiar with the capability implementation plans. Unfortunately, this assumption may 
be incorrect. The committee that participated in the collaborative process also serves as 
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the committee with direct responsibility for implementing the Hazardous 
Materials/WMD Response and Decontamination capability. This committee provided 
assessment ratings that were opposite those of the fire discipline, in relation to the 
number of measures above or below average mean perfonnance measure. This finding is 
the same finding as was discovered in the Mass Surge assessment when it was examined 
by discipline. 

The finding — that the aggregate ratings of a group of individuals from a single 
discipline are different from the ratings resulting from a collaborative effort of similar 
individuals — is a significant finding. It may be a function of positive interaction and 
learning on the part of the collaborative group. This is the experience of the an 
assessment group in Minnesota who had subgroups complete the assessment survey, then 
come together as a group to discuss the subgroup results and reach a consensus rating. 
The description provided by members of this group was that the consensus effort was a 
learning experience and the changed scores reflected knowledge learned about 
capabilities in the consensus process. 46 

If this description is the cause, then the knowledge of the experts selected to 
perform the assessment is called into question. If the knowledge about the capability 
assessed is advanced in the assessment process, then gained knowledge is a positive 
outcome and satisfies a goal of the assessment process. This finding is also indicative of 
a lack of knowledge and the results of such an assessment are of questionable use to 
develop strategic plans to close capability gaps. 

The assessment disciplines and process groups’ ratings indicate the potential for 
inconsistent results when using the assessment tool. This analysis has been peeling back 
the layers of the assessment to scrutinize the data. The data was first analyzed by the 
assessment process, which showed that groups may be able to identify a trend in 
capability status. The data at the discipline level began to show scoring bias based on 
discipline responsibility. The data at the discipline level also showed disagreement 
within disciplines about the status of capabilities. 

46 Natasha Beluch, “Meeting Notes, May 16, 2007.” Pilot Capabilities Assessment Project (DHS: 
Washington, DC: Mayl6, 2007), 2. 
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G. 


CONCLUSION 


The analysis of the data suggests further exploration of the ability of the 
homeland security discipline to conduct a self assessment. If planning efforts at closing 
capability gaps are to be informed by the results of a self assessment, then the potential 
for errors in future planning efforts exists. Through observations and interviews with 
survey participants, a trend emerges as to why there are differing opinions on capability 
status. Discovery of this trend was found when looking for data on the benefits of the 
capability assessment process. 

The following example suggests two problems. The first suggestion is that the 
subject-matter experts assembled to assess the capabilities did not have the expertise to 
conduct the assessment. The second suggestion is that the assessment process produced 
collaboration in an effort to solve a homeland security problem. 

In this example, a senior emergency management planner agreed to participate in 
Assessment Group 1. Once the planner received the assessment, he did not feel 
comfortable providing answers to the Mass Surge assessment because of his lack of 
expertise in the subject. Instead of returning the survey incomplete, he passed it to an 
Urban Area Security Initiative Program Manager for the Health Emergency Support 
Function. This program manger did not feel comfortable performing the assessment and 
passed it on to the Council of Governments Medical Surge Hospitals and Health 
Subcommittee for completion during a conference call. 47 The subcommittee ultimately 
completed the survey by consensus and thus became Assessment Group 4 for the Mass 
Surge Capability assessment. 

This example suggests a problem with the knowledge related to specific 
homeland security capability by a senior emergency management official. This lack of 
knowledge was further compounded when the official passed the assessment on to a 
program manager whom he mistakenly thought had the requisite knowledge to complete 
the assessment. The program manager also lacked the knowledge, but passed on the 
assessment to a group that she thought had the knowledge to complete the assessment. 

47 Email from Participant 8 to author, March 28, 2007. 
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If we could use the results of the completed assessment to show that this group 
reached a valid conclusion about the status of the measured capability, we could 
ultimately count the outcome of this example as a success, but the data suggests a broad 
diversity of opinions exists and that the assessment process has not demonstrated the 
ability to capture that diversity. As a result, that failure suggests an example in which a 
lack of knowledge, compounded at each level of responsibility, produced results that may 
be mistaken as valid. 

On the positive side, this example demonstrated collaborative effort expanding 
from a single assessor to a group, in an attempt to develop an understanding of the 
capability. This type of collaboration is necessary to ensure a diversity of opinion and, 
once knowledgeable assessment group members are identified, will help ensure a valid 
result. 

Did the different processes used affect the outcome? The evidence is notclear. 
Figures 10 and 11 exhibit the scoring by groups for the capabilities. The charts show the 
wide diversity of opinions on some measures and near agreement on others. These charts 
also suggest a trend in which the groups using the aggregated assessment process result in 
similar ratings (Groups 1 and 2) and the consensus groups provide either higher or lower 
ratings. This trending is interesting, but alone it does not seem to indicate a significant 
difference of opinion about the status of the capability. 
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V. CONCLUSION 


The Unknown 

As we know, 

There are known knowns. 

There are things we know we know. 

We also know 

There are known unknowns. 

That is to say 

We know there are some things 
We do not know. 

But there are also unknown unknowns. 

The ones we don't know 
We don't know. 

—February 12, 2002, Department of Defense news briefing 

Donald Rumsfeld 48 


A. INTRODUCTION 

This thesis explores the effect on outcomes when altering the process by which a 
specific assessment methodology is employed. The research question became: A DHS 
working group has proposed a methodology for performing capability assessments using 
local subject-matter experts. Can the portion of the assessment methodology designed to 
create collaborative opportunity be altered without affecting assessment outputs, while 
still creating the desired additional benefits of enabling and improving collaboration 
among homeland security professionals? 

The small scale of the study limits how the results of the assessments can be 
interpreted. By themselves, the surveys mean little. When combined with the interviews 
of assessment team participants, however, potentially important insights into the 
understanding of capabilities at the local level are suggested. The findings and 
recommendations in this chapter discuss this issue in more detail. 


48 Hart Seely, “The Poetry of D. H. Rumsfeld,” Slate, April 2, 2003, 
http://www.slate.com/id/2081042/, accessed July 19, 2007 
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B. FINDINGS 

This thesis set out to discover the level of effect that three different processes, 
used to conduct a self assessment, had on the data collected. The hypothesis was that the 
processes would not play a key role in the results, and that data collected from different 
processes would be comparable to the other. The thesis further hypothesized that the 
different processes would produce different levels of collaboration among the 
participants, and that one process would be identified as having provided more additional 
benefits than the others. 

The results of the comparative case study differed from the hypothesis and 
suggest that the assumption that “self-assessment” is a reliable and valid methodology for 
performing homeland security capabilities assessment may be incorrect. Two points that 
support these findings. First, the experts — the “self’ — are not sufficiently 
knowledgeable of the capabilities to perform the self assessment, and the process used in 
the assessment does not matter. In each of the three processes used in the comparative 
case study, group think reduction to the established mean appeared to be accepted. These 
findings suggest that strategy for the national assessment should be adjusted to fit the 
reality. If we assume we can do an assessment, and the assessment is faulty, what has 
changed in our assessment strategy since the last assessment? The answer is “nothing.” 
As a result, homeland security may be doing more harm than good by providing an 
illusion of preparedness. The illusion of preparedness — based on imprecise 
measurements — causes our community to make investments to close gaps that 
ultimately may not have existed. This becomes a missed opportunity that affects our 
national preparedness. 

The data suggests that the homeland security community in the National Capitol 
Region includes very few people who can claim and demonstrate the knowledge 
necessary to assess the status of regional multi-discipline efforts to implement the NPG. 
If true, this finding is particularly surprising since many of the participants in the surveys 
are those responsible for such efforts. This lack of subject-matter expertise casts doubt on 
the ability of a self-assessment tool to capture an accurate snapshot of capability status. 
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There is reason to believe that this problem is national in scope. Preliminary 
results from a pilot test of the self-assessment process conducted in Minnesota found a 
similar problem in expertise of the assessment group. In Minnesota, over one-third of 
emergency managers in the state are new. These members felt they could use the 
assessment process to educate the emergency managers. 49 This evidence appears to 
support the finding that a lack of expertise exists to assess capabilities. 

A related finding is that members of the homeland security community were 
willing to use the results from this tainted survey process even after they understood the 
problems with the data. Members felt they could develop future plans because they felt 
the tainted results were “about right and when viewed in comparison with other 
capability results gave an impression of status.” 50 , 51 This finding may be more important 
than the finding that the members lacked the knowledge to conduct the assessment; it 
shows they do not understand the importance of assessment. This finding indicates a 
need to develop a strategy for capability-based planning education to reinforce the 
foundation of our homeland security strategies. 

Additional evidence, that the problem may be larger than the region, comes from 
a recent report issued by John Hopkins University. UASI homeland security 
professionals were surveyed about compliance with an understanding of HSPD 8. 
Recommendations to improve HSPD 8 compliance, based on data collected in the survey, 
called for enhanced regulation of homeland security grants and improved risk assessment, 
but notably failed to include recommendations pertaining to capability assessment. 52 
This omission represents the core of the assessment problem: The discipline is immature 
and its members need guidance and mentoring to institutionalize preparedness. 

If a theme is suggested from the analysis, it can be best defined as a disconnection 
between the goal of a capability assessment process and the relationship of capabilities to 

49 Lauren Fernandez email to author, Washington, DC, June 11, 2007. 

50 Interview of Participant 3 by author, Washington, DC, June 5, 2007. 

51 Interview of Participant 5 by author, Washington, DC, June 6, 2007. 

52 Amber Gisriel, William Grant, Brian Lee, Steve Fruchtman, “HSPD-8 Compliance by UASI 
Jurisdictions (John Hopkins University May 2007), 43 
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local efforts. The inconsistent data on capability status produced by the surveys seems to 
indicate a lack of knowledge by local homeland security leaders. This finding is 
probably true; members of the assessment groups have been promoted within their 
organizations through civil service processes that infer a level of expertise within the 
traditional knowledge areas of their organizations. There is not a discipline within this 
regional community of homeland security capability officials and, as a result, no 
comparable level of hierarchical expertise can be inferred. 

This thesis did not specifically examine what local officials understood about the 
capabilities-based planning requirements of HSPD 8 or whether they considered the TCL 
an appropriate building block for preparedness. Therefore, a lack of knowledge by local 
officials cannot be assumed. It may be safe to assume though, that if local leaders were 
placing the same priorities upon the capabilities being measured as does the Department 
of Homeland Security, it would follow that they would have a better understanding of 
those capabilities. It is possible that the findings of this comparative case study are the 
result of this disconnection in priorities. 

The findings related to collaboration were not as hypothesized, and they support a 
possible theory that members of the homeland security community are focused on 
different issues than what is being measured in the capabilities. Each assessment process 
produced demonstrable collaborative efforts; members sought information about 
capability status from contacts they believed had the expertise to provide capability status 
information. This indicates that there is an established network in the homeland security 
community. This network would not exist without reason. 

Assessment Group 1 provided an example of this collaboration using existing 
networks. The design of this group suggested that members work alone and complete the 
process with little or no collaborative effort. Instead, members of this group sought out 
other members to discuss capabilities before assigning a rating. 53 The group knew where 
to look for expertise and had the desire, relationships and authority to have others 


53 Interview of Participant 3 by author, Washington, DC, June 5, 2007. 
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perform the assessment. This is not necessarily an example of lack of knowledge, but an 
example of the confusion in the measurement process. 

An example of the desire for collaboration of the assessment groups supported a 
need for collaboration. A majority of survey participants stated that the inclusion of a 
group process in the self-assessment tool provided the opportunity to discover unknown 
information about a capability and would improve their effort to develop solutions and 
priorities to close a capability gap. A hazardous materials response official found that the 
discussion allowed better understanding of other disciplines’ needs to create capability to 
meet the demands of a measure. 54 A police official felt that the opportunity for 
collaboration was an opportunity to engage and educate support functions about their 
roles in creating a capability. 55 

To summarize the results of this comparative case study, the most important 
suggestion is that local officials cannot use this assessment tool to provide the snapshot of 
capability status desired by the Department of Homeland Security. This is not necessarily 
because of faults in the assessment tool, but because of the knowledge base of the local 
officials. An examination of the results suggests an undercurrent of homeland security 
activities that the assessment tool does not capture: Local officials are focused on 
homeland security issues that are not the same as the assessment tool measures, and they 
lack an appreciation for the significance of assessment. 

C. RECOMMENDATIONS 

The comparative case study conducted in this thesis suggests insight into the 
limits of the capability-assessment process in the homeland security discipline. The 
findings suggest an understanding of the status of a capability was unlikely to be 
gained directly from this assessment methodology. 

These results suggest a need to change strategy in our homeland security 
capabilities-assessment processes. Current assessment strategies are designed to 

54 interview of Participant 13 by author, Washington, DC, June 8, 2007. 

55 Interview of Participant 14 by author, Washington, DC, June 5, 2007. 
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document organizational accomplishments, but do little to reveal the effect of those 
accomplishments. An example of an area that has an existing strategy that may be 
effective is in the fire service. The consensus standards issued by the National Fire 
Protection Association create a national model of minimum fire protection standards 
calibrated to community size and resources. Through minimum training standards, local 
fire chiefs and firefighters develop knowledge of these standards. Because the 
knowledge of the firefighting capability is standardized through an accepted consensus 
process, the standards have validity that the current assessment process lacks. 

These standards create levels of understanding that are national in scope. For 
example, fire fighters with basic fire training anywhere in the county learn the same fire 
prevention strategies. 56 Fire officers around the country receive the same information 
about response times and safety standards. 57 This widespread knowledge is missing from 
the homeland security discipline. The language of the discipline has not yet been fully 
developed. 

A new assessment strategy would place a higher priority on the understanding of 
homeland security capability than on the assessment of homeland security capabilities. 
The new strategy should focus on identifying what is working within homeland security. 
The needed changes require the development of concepts for asking the right questions, 
questions that will create knowledge of local priority-setting processes and planning 
efforts. 

The results of these assessments could then be used to develop a national picture 
of local capabilities using a different matrix than the NPG. A new matrix would provide 
an overview of ah preparedness efforts, not just the ones associated with HSPD 8. This is 
necessary because the federal direction about preparedness activities is conflicted. 

Not ah federal planning efforts align with the NPG. Specifically, HSPD 5 and the 
National Response Plan designate additional capabilities, planning assumptions or 

56 NFPA 1001, Standard for Fire Fighter Professional Qualifications, 2008 Edition, Annex A 
Explanatory Material, 6.5.2 (NFPA< Quincy, MA). 

57 NFPA 1021, Standard for Fire Officer Professional Qualifications, 2003 Edition, Chapter 7.6 
Emergency Services Delivery and 7.7 Flealth and Safety (NFPA: Quincy, MA). 
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separate functions that create confusion when local authorities try to detennine 
implementation priorities for capability-based planning efforts. This misalignment may 
help to explain why local officials have developed systems that are not captured in the 
capability assessment. 

For example, HSPD 5 addresses two of the elements listed on the TCL: on-site 
and emergency operations center management. The HSPD 5 scenarios match only 5 of 
the 15 listed under the TCL or HSPD 8: hurricane, earthquake, hazardous materials and 
nuclear. 58 The National Response Plan designates responsibilities among federal 
agencies for federal response to significant events. The NRP designates 15 Emergency 
Support Functions to accomplish response tasks. “Four of the 15 ESF groupings do not 
align with the elements of the TCL: transportation, public works and engineering, energy 
and external affairs.” 59 

To develop the necessary snapshot of capability status, a new assessment strategy 
will have to assess more than HSPD 8 and the TCL. The measurement must include all 
preparedness efforts that local authorities are trying to implement. These efforts will 
provide the needed data to detennine capability status. 

1. Leadership 

This thesis has identified problems with the levels of homeland security 
knowledge and recommended improved education in capabilities-based planning for 
homeland security leaders. The thesis has also argued that the NPG, when used as a 
measuring stick for these professionals, may be the wrong measure. 

This is similar to the situation Bratton faced when he took over the New York 
City Police Department. His commanders did not believe there were crime problems in 
the subway system because the statistics told them only 3 percent of the city’s major 
crimes took place in the subway. Bratton felt the commanders’ assessment of crime was 

58 Chris Bellavita, Complex Adaptive Lecture, July 17, 2007, quoting White House, Homeland 
Security Presidential Directive/HSPD 5 (Washington, DC: The White House, February 28, 2003), 22. 

59 Bellavita Lecture, July 17, 2007, quoting National Response Plan (Washington, DC: Department of 
Homeland Security, March 2004): 19. 
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incorrect. Bratton heard the complaints of citizens about conditions in the subways and 
dispatched his commanders to ride the subway. In a matter of weeks, police strategy 
changed to focus on reducing crime on the subways, because the police commanders 
recognized the problems with their previous assessments and adjusted their policing 
strategies accordingly. 60 

By viewing the tested capabilities-assessment process as crime statistics in the 
subway system in this example, we can see that we are maintaining the status quo in 
regards to our knowledge. If we require our leadership to participate in an educational 
assessment process that focuses on understanding the existing capabilities and needs, we 
should be able to develop leadership momentum for changes to the methodologies we use 
to examine capabilities. Riding the subway was the educational assessment process for 
the police commanders. 

The assessment process becomes the tipping point. Leaders who cannot show 
knowledge of their capabilities — and defend those capabilities as being appropriate to 
their local needs — should be replaced with leaders who can. The key to this tipping 
point is that it is tied to the leaders’ understanding of their capabilities as related to local 
needs, not to the NPG. 

2. New Assessment Tactics 

The development of tactics for gaining an understanding of what local homeland 
security efforts are, and how they are developing, can be very simple and straightforward. 
A large, relatively untapped body of capability status evidence exists. This evidence is 
related to actual emergencies and all levels of response to the incident. There is also a 
large body of evidence related to exercises. 

A new assessment tactic could the comparison of actions related to a past incident 
to the same incident scenario today. The actual incident could be studied through after- 


60 W. Chan Kim and Renee Mauborgne, Blue Ocean Strategy (Boston, MA: Harvard University Press, 
2005), 153. 
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action reports and interviews with key decision makers. The current scenario can be 
assessed through interviews at multiple levels within an organization. The result is to 
understand the effect of an investment. 

For instance, the Arlington County After Action Report on the 9/11 Pentagon 
Attack can be used as a basis for examining an actual response. Asking the local police 
and fire officials, “What would be different if the attack were to occur today?” the 
discussion would continue until they gained an understanding of the effect of any added 
capabilities. 

For example, would the dollars invested in urban search and rescue teams have 
saved any lives in the same scenario? If the answer is no, then the net effect of the 
investment is zero. If the answer is yes, then there is a gain from the investment. In 
either case, questions about why the urban search and rescue investment received funding 
and how the investment was implemented will provide useful data. This data will allow a 
broader examination of influences on the homeland security decisions of local 
governments. 

To examine a capability that does not have an incident parallel, we can use 
exercise data. A comparative case study of exercises and response polices to radioactive 
dirty bombs may reveal that the baseline policy for a region on 9/11 would have 
consisted of rapid response by all disciplines to perform rescue, removal and 
decontamination of potentially exposed persons. A comparative case study of 
investments related to radiation response may show multiple programs to improve 
personal protection, detection and decontamination. 

A comparative case study of recent exercises and current response polices may 
indicate that the effect of response polices remains the same as on 9/11: rapid response by 
all disciplines to perform rescue, removal and decontamination of potentially exposed 
persons. At first blush, it would seem that no gain has been accomplished through the 
investments because the effect of the response is the same before and after the 
investment. 
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Questioning of members in various disciplines should take place to identify other 
areas of benefit from the investments. Arguments can be made that the investments will 
result in rapid identification of a radiation dirty bomb incident and that this early 
notification will allow notification to hospitals, utilities and transportation disciplines to 
protect critical infrastructure. Early notification will cause responders to take specific 
actions that will improve their safety, and such increased detection capability creates 
opportunity for incident prevention. 

These examinations create opportunities for tipping point leadership. The 
examination creates opportunity for self examination among leaders. For example, if a 
fire chief has invested heavily in Urban Search and Rescue and, through this process, sees 
that the investment did not affect the outcome of an incident, then the fire chief has an 
opportunity to change the way he will evaluate future investments. Having to defend the 
effect of investments in radiation-detection capabilities can cause the fire chief to develop 
a keener understanding of the complexities of a radiation response. This knowledge can 
change behavior and become a tipping point for a better understanding of homeland 
security investments. Through improved investing, the leader can make changes that 
increase preparedness. Leaders who continue to make investments that would not 
improve an outcome will soon find themselves and their organizations left behind in the 
funding processes. 

3. Areas for Future Research 

This thesis identified an area of strength within the capability assessment process. 
This strength is the desire and willingness of the homeland security community to use 
collaborative efforts to solve problems. The recommendation made in this thesis is to 
utilize areas of strength in homeland security and use those strengths as the basis for 
future efforts to learn about homeland security capabilities. 

Continuing to create the academic field of homeland security is one of the keys to 
implementing an understanding of homeland security and establishing national standards 
and policies for homeland security. The tactic described to compare and contrast past 
and current capability requires expertise levels sufficient to develop an understanding of 

58 



the effects of changes in capabilities. This thesis has shown there are problems with 
current levels of expertise; overcoming this issue will be challenging. 

Additionally, an improved understanding of effects-based planning related to 
homeland security is needed. The military is using effects-based planning for 
warfighting capabilities . 61 These efforts should be examined for transferable knowledge 
to the homeland security discipline. 

D. CONCLUSION 

This thesis evolved because of a perception that the capability assessment 
requirements being heaped on local government were not adding value to homeland 
security efforts. The evolution of possible solutions to this problem sought to add value 
and make improvements to existing strategies in capabilities assessments. Ultimately a 
promising tool was developed and tested. The analysis of the testing suggests that the 
tool may not function as designed. This is not necessarily because the tool was 
ineffective, but because existing homeland security strategies have not focused on 
developing the understanding of local capabilities and needs necessary to assess those 
capabilities. 

The Department of Homeland Security stands to gain from investment in 
understanding of total capability by local government through participation in this 
process. The migration of perspective — from national capability to a regional capability 
to local capability — is necessary to make substantial and lasting gains in homeland 
security preparedness. This perspective is what is missing from current efforts and we 
should focus on causing change for future efforts. 


61 Paul Davis and Gail Zellman, Effects Based Operations (Santa Monica, CA: RAND, 2001), 1. 
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APPENDIX A: MEASURES - MASS SURGE CAPABILITY 


Measure 

Description 

1 

Our region's plans specify the criteria and process for activating Medical Surge procedures. 

2 

Our region’s plans for Medical Surge are based on a formal assessment of risks and 
vulnerabilities. 

3 

Our region's plans address the use of existing facilities in support of Medical Surge (e.g., 
hospitals, clinics, extended care facilities). 

4 

Our region’s plans address the identification and setting up of additional facilities (e.g., 
provision of personnel, equipment, pharmaceuticals). 

5 

Our region's plans address cultural characteristics of populations to be treated (e.g., religious 
needs, language barriers). 

6 

Our region's plans address the treatment requirements of special needs populations. 

7 

Our region's plans address patient and resource transportation (e.g., identification and 
availability of traditional and non-traditional resources). 

8 

Our region's plans address facility based evacuation (e.g., identification of receiving facilities, 
coordination of transportation assets). 

9 

Our region's plans address the treatment of Medical Surge response personnel, site staff, and 
their families (e.g., medical needs, stress management strategies). 

10 

Our region’s plans address public information and communications requirements relevant to 
Medical Surge (e.g., dissemination of accurate, timely, accessible information to public, 
media, support agencies). 

11 

Nonprofit organizations/NGO’s are actively engaged in your region’s development of plans for 
Medical Surge (e.g., the American Red Cross, Salvation Army, faith-based organizations). 

12 

Our region's emergency transport and tracking systems are interoperable with national and 
Department of Defense systems. 

13 

Our region’s supply inventories and reserves are adequate to support of Medical Surge. 

14 

Our region maintains databases that track the status of resources (e.g., medications, medical 
professionals) available to support Medical Surge. 

15 

Our region currently has (or has access to) the personnel it needs to support Medical Surge 
(e.g., medical service providers, patient transporters, security personnel). 
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16 

Our region has a robust program for recruiting volunteers to support Medical Surge. 

17 

Our region has developed a method for managing spontaneous volunteers and donated 
resources in support of Medical Surge (That is, is there a system to intake volunteers, assign 
responsibilities and direct resources where they are most needed). 

18 

Our region currently has (or has access to) the equipment it needs to support Medical Surge 
(e.g., medical equipment, transportation vehicles). 

19 

Our region is able to ensure safety and security at the facilities it intends to use to support 
Medical Surge (That is, can your region determine and ensure the structural integrity, 
capability, and physical security of its Medical Surge facilities). 

20 

Our region is able to ensure the self-sufficiency of the facilities it intends to use to support 
Medical Surge (e.g., with respect to communications, power). 

21 

Our region has the information resources required to project the demand for Medical Surge 
(e.g., how many people will need treatment, how long it will take to secure facilities). 

22 

Our region has addressed the credentialing requirements of Medical Surge. 

23 

Our region have scalable patient tracking systems. 

24 

Our region has exercised its ability to implement Medical Surge (e.g., the provision of 
medication, follow-up care, securing of facilities). 
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APPENDIX B: MEASURES - HAZARDOUS MATERIALS / WMD 
RESPONSE AND DECONTAMINATION CAPABILITY 


Measure 

Description 

1 

Our region’s plans specify the criteria for activating WMD/Hazardous Materials Response and 
Decontamination procedures. 

2 

Our region’s plans for WMD/Hazardous Materials Response and Decontamination are based on a formal 
assessment of risks and vulnerabilities. 

3 

Our region’s plans for WMD/Hazardous Materials Response and Decontamination address responder 
safety (e.g., accurate identification of hazards, appropriately certified responders, proper PPE). 

4 

Our region’s public safety and HazMat personnel are trained at various levels (e.g., hazardous awareness, 
hazardous materials operations, hazmat technician, hazmat specialist, hazardous materials management). 

5 

Our region’s HazMat personnel are equipped and trained for weather prediction and hazard plume 
modeling. 

6 

Our region’s plans for WMD/Hazardous Materials Response and Decontamination address substance 
identification equipment (e.g., bases, vapors, liquids, solids, biologicals like white powder). 

7 

Our region’s plans for WMD/Hazardous Materials Response and Decontamination address personnel 
needs (e.g., work/rest cycles, medical, psychological, financial assistance, etc). 

8 

Our region’s plans address information sharing requirements relevant to WMD/Hazardous Materials 
Response and Decontamination (e.g., dissemination of accurate, timely, accessible information to public, 
media, support agencies, etc.). 

9 

Our region currently has (or has access to) the personnel it needs to support WMD/Hazardous Materials 
Response and Decontamination (e.g., safety officers, HazMat personnel, medical specialists). 

10 

Our region currently has (or has access to) the equipment it needs to support WMD/Hazardous Materials 
Response and Decontamination (e.g., PPE, medical equipment, WMD detectors). 

11 

Our region has information tracking systems relevant to WMD/Hazardous Materials Response and 
Decontamination (e.g., hot zones locations, real-time hazard databases, decontaminated persons 
updates). 

12 

Our region is able to ensure the safety and security at the scene and support facilities it intends to use for 
WMD/Hazardous Materials Response and Decontamination (That is, can your region determine and 
ensure the structural integrity, capability, and physical security of its HazMat Management facilities.). 

13 

Our region is able to ensure the self-sufficiency of its WMD/Hazardous Materials Response and 
Decontamination operations (e.g., with respect to communications, power). 

14 

Our region’s plans address the demobilization of WMD/Hazardous Materials Response and 

Decontamination (e.g., debrief personnel, repackage equipment). 

15 

To what extent has your region exercised its ability to implement WMD/Hazardous Materials Response 
and Decontamination procedures (e.g., exercised response to pre-identified hazards, decontaminated 
victims). 
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